feat: Add hosted evals options. #303
Open
+727
−11
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Medium Risk
Adds a new hosted execution path that creates/polls evaluations via platform APIs and can prompt to publish environments, which may change user workflows and depends on API behavior/log streaming.
Overview
Adds a new
--hostedmode toprime eval(and the deprecatedprime env eval) that runs evaluations on the Prime Intellect platform instead of locally. This path requires anowner/nameenvironment slug, resolves the environment ID from the hub (optionally prompting toprime env pushif only a local unpublished env is found), submits a hosted evaluation request, then polls status and optionally streams cleaned logs until completion.Introduces
prime_cli.utils.hosted_evalwith the hosted-eval API calls, polling/log-streaming UI, and log-cleaning helpers, plus unit tests covering ANSI/progress-bar filtering and log streaming overlap behavior.Written by Cursor Bugbot for commit c1335d1. This will update automatically on new commits. Configure here.